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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1)H Responsive to communication(s) filed on 27 November 2001 . 
2a)D This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) E3 Claim(s) 1-19 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) I3 Claim(s) 7-79 is/are rejected. 

7) D Claim(s) is/are objected to. 

&*)□ Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9)13 The specification is objected to by the Examiner. 

10) [3 The drawing(s) filed on 27 November 2001 is/are: a)S accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

'\2)M Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
aM All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 



Claim Objections 



1 . Claim 8 is objected to because of the following informality: the word "method" is used 
rather than "system" (Claim 8, In. 3) in a system claim (Claim 8, In. 1). Appropriate correction 
is required. 



application or patent, or to a publication is improper. Instances in the specification include, but 
are not limited to: 

- EPO 00116861.6 (Specification: p. 11, In. 22) 

- Wilcox et al., "Segmentation of Speech Using Speaker Identification", 1994, ICASSP 
(Specification: p. 13, Ins. 21-22; p. 16, Ins. 8-10); 

- Montacie et al., "A Silence/Noise/Music Speech Splitting Algorithm", 1998, ICSLP 
(Specification: p. 31, Ins. 22-24; p. 16, Ins. 10-12); 

- Muthusumy et al., "Automatic Language Identification : A Review/Tutorial", 1994, IEEE 
Signal Processing; 

- Farach et al., "Sparse Dynamic Programming of Evolutionary-Tree Comparison", 1997, 
SLAM Journal of Computing; and 

- Melamed, "A Geometric Approach to Mapping Bitext Correspondence", 1996, EMNLP 
(p. 17,1ns. 6-7; p. 19,1ns. 17-19). 



Specification 



2. 



The incorporation of essential material in the specification by reference to a foreign 
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Applicant is required to amend the disclosure to include the material incorporated by reference. 
The amendment must be accompanied by an affidavit or declaration executed by the applicant, 
or a practitioner representing the applicant, stating that the amendatory material consists of the 
same material incorporated by reference in the referencing application. See In re Hawkins, 486 
F.2d 569, 179 USPQ 157 (CCPA 1973); In re Hawkins, 486 F.2d 579, 179 USPQ 163 (CCPA 
1973); and In re Hawkins, 486 F.2d 577, 179 USPQ 167 (CCPA 1973). 

Additionally, improper attempts to incorporate essential subject matter by reference of a 
U.S. patent or application which itself incorporates essential subject matter by reference into this 
application is improper because. See MPEP § 608.01(p) Instances include, but are not limited 
to: 

- U.S. Patent Application No. 09/447,871 (Specification: p. 6, In. 4; p. 12, In. 20; p. 17, 
In. 28; p. 18, In. 2; p. 18, In. 8). 

Correction is required. 

3. The use of the trademarks have been noted in this application. Examples include, but are 
not limited to: 

- IBM VIAVOICE (TM) (Specification: p. 8, In. 1); 

- LATEX (TM) (Specification: p. 14, In. 4); and 

- MICROSOFT WORD (TM) (Specification: p. 14, In. 22). 

Trademarks should be capitalized wherever they appear and be accompanied by the generic 
terminology. 
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Although the use of trademarks is permissible in patent applications, the proprietary 
nature of the marks should be respected and every effort made to prevent their use in any manner 
which might adversely affect their validity as trademarks. Correction is required. 

Claim Rejections - 35 USC § 102 

4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 

basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

5. Claims 1 and 13 are rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent No. 5,515,490 issued to Buchanan et al. (hereafter Buchanan '490). 

Claim 1: 

Regarding Claim 1, Buchanan '490 discloses an automatic temporal formatter for 
synchronizing multimedia data streams such as video, audio, and text (e.g. subtitles). 
Specifically, Buchanan '490 discloses: a computer-based method of synchronizing a realization 
of a media (Buchanan '490: Abstract) stream having a first representation synchronized with 
said realization, and at least one second representation (Buchanan '490: col. 57, Ins. 11-13), said 
method comprising: 

- determining structure information for said first representation and said at least one second 
representation (Buchanan '490: col. 23, Ins. 59-65; col. 57, Ins. 20-30); 
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- determining structure association information between said first representation and said at 
least one second representation (Buchanan '490: col. 23, In. 66 to col. 24, In. 10; col. 57, 
Ins. 31-50); and 

- synchronizing said at least one second representation with said first synchronized 
representation and said realization using said structure association information (Buchanan 
'490: col. 24, Ins. 11-15; col. 57, Ins. 51-63; col. 58, Ins. 9-23). 

Claim 13: 

Examiner notes that Claim 13 is the apparatus embodiment of Claim 1 and is rejected on 
the same basis. 

Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

7. Claims 2-3 and 14-15 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Buchanan '490 in view of the publication, "Synchronization Relation Tree : A model for 
Temporal Synchronization in Multimedia Presentation", by Kim et al. published as Technical 
Report TR92-42, by the Dept. of Computer Science, Univ. of Minnesota, 1992 (hereafter Kim 
'92). 

Claim 2: 
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Regarding Claim 2, Buchanan '490 discloses all the limitations of Claim 1 (supra). 
Buchanan '490 additionally discloses: said step of deterrnining structure information further 
comprising: analyzing said structure information of said first and said at least one second 
representation (Buchanan '490: col. 23, Ins.. 59-65; col. 57, Ins. 20-30). Furthermore, Buchanan 
'490 discloses providing a stream of temporal data (Buchanan '490: col. 23, Ins. 59-65; col. 3, 
Ins. 40-47, note that data provided continuously over runtime reads on a stream). However, 
Buchanan '490 does not explicitly disclose: the stream of temporal data comprised of tree 
locators. 

Kim '92 discloses a synchronization relation tree (Kim '92: Abstract). (Note that a data 
structure that contains pointers to data corresponding to the nodes rather than the data itself reads 
on tree locators). 

It would have been obvious to a person having ordinary skill in the art to apply the 
synchronization relation tree of Kim '92 to the automatic temporal formatter of Buchanan '490. 
The motivation to combine is suggested by Kim '92 which discloses: the synchronization 
relation tree provides for both "temporal relationship consistency" and "dynamic schedule 
completion" and further is better suited for an object-oriented implementation (Kim '92: p. 3, In. 
38 to p. 4, In. 3). 
Claim 3: 

Regarding Claim 3, Buchanan '490 and Kim '92 in combination disclose all the 
limitations of Claim 2 (supra). Further note that Buchanan '490 and Kim '92 in combination 
disclose: aligning said determined structure information of said first representation and said at 
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least one second representation (Buchanan '490: col. 24, Ins. 11-15; col. 57, Ins. 51-63; col. 58, 
Ins. 9-23). 
Claims 14-15: 

Examiner notes that Claims 14-15 are the apparatus embodiment of Claims 2-3 
respectively and are rejected on the same basis. 

8. Claims 4-5, and 16-17 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Buchanan '490 and Kim '92 in view of the publication, "Cooperative Use of MHEG-5 and 
HyTime", by Rutledge et al., published by Proceedings of Hypertext and Hypermedia, 1997 
(hereafter Rutledge '97). 
Claim 4: 

Regarding Claim 4, Buchanan '490 and Kim '92 in combination disclose all the 
limitations of Claim 3 (supra). Buchanan '490 and Kim '92 in combination further disclose: 
wherein said realization comprises at least one version of content, said method further 
comprising: aligning said at least one version of content with said first representation (Buchanan 
'490: col. 24, Ins. 11-15; col. 57, Ins. 51-63; col. 58, Ins. 9-23). However, Buchanan '490 and 
Kim '92 in combination do not explicitly disclose: to produce a web of relations between said at 
least one version of content and said first representation. 

Rutledge '97 discloses MHEG-5 and HyTime (Hypermedia/Time-based Structuring 
Language): producing a web of relations (Rutledge '97: Section 2, titled "Standards for 
Hypermedia", second paragraph). 
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It would have been obvious to a person having ordinary skill in the art to apply the 
HyTime language of Rutledge '97 to the Buchanan '490 and Kim '92 combination. The 
motivation to combine is suggested by Rutledge '97 which discloses: HyTime especially in 
cooperation with MHEG-5 provides a particularly advantageous combination for the encoding of 
hypermedia (and multimedia) presentations (Rutledge '97: Abstract). 
Claim 5: 

Regarding Claim 5, Buchanan '490, Kim '92, and Rutledge '97 in combination disclose 
all the limitations of Claim 4 (supra). Further note that Buchanan '490, Kim '92, and Rutledge 
'97 additionally disclose: aligning said at least one version of content with said first 
representation produces a web of relations between a structural view of said at least one version 
of content and said first representation (Buchanan '490: col. 24, Ins. 11-15; col. 57, Ins. 51-63; 
col. 58, Ins. 9-23; Rutledge '97: Section 2, titled "Standards for Hypermedia", second 
paragraph). 
Claims 16-17: 

Examiner notes that Claims 16-17 are the apparatus embodiment of Claims 4-5 
respectively and are rejected on the same basis. 

9. Claims 6-7 and 18-19 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Buchanan '490 and Kim '92 in view of the publication, "Using the Strategy Design Pattern to 
Compose Reliable Distributed Protocols", by Garbinato et al. published by the USENIX 
Conference on Object-Oriented Technologies and Systems, 1997 (hereafter Garbinato '97), and 
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in further view of U.S. Patent No. 5,731,847 issued to Tsukagoshi et al. (hereafter Tsukagoshi 
'847). 

Claims 6-7: 

Regarding Claim 6, Buchanan '490 and Kim '92 in combination disclose all the 
limitations of Claim 3 (supra). Further note that Buchanan '490 and Kim '92 in combination 
disclose: aligning media streams (Buchanan '490: col. 24, Ins. 11-15; col. 57, Ins. 51-63; col. 58, 
Ins. 9-23). However, Buchanan '490 and Kim '92 in combination do not explicitly disclose: 

- (Claim 6) aligning an audio stream specified by said media stream with an audio structure 
corresponding to said audio stream; or 

- (Claim 7) aligning a text stream specified by said media stream with a text structure 
corresponding to said text stream 

Tsukagoshi '847 discloses an encoder and decoder of subtitle information. Specifically, 
Tsukagoshi '847 discloses: 

- (Claim 6) aligning an audio stream specified by said media stream (Tsukagoshi '847: 
col. 11, Ins. 45-50 and Alleva '197: col. 13, Ins. 40-46). Note that while analysis of an 
audio stream under Tsukagoshi '847 is optional, the combination of Alleva '197 to 
Tsukagoshi '847 requires the generation and subsequent alignment of an audio structure 
from an audio stream 

(Claim 7) aligning a text stream specified by said media stream (Tsukagoshi '847: col. 
11,1ns. 28-35). 
However, Tsukagoshi '847 does not explicitly disclose: 

- (Claim 6) aligning with an audio structure corresponding to said audio stream; or 
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- (Claim 7) aligning with a text structure corresponding to said text stream. 

Garbinato '97 discloses the well-known Strategy design pattern. Specifically, Buchanan 
'97 discloses that objects designed to handle distinct types of data and/or interactions are to be 
distinct via the Strategy design pattern (Garbinato '97: p. 1, col. 2, Ins. 14-27). 

It would have been obvious to a person to apply the substitute the automatic temporal 
formatter of Buchanan '490 for the rate controller with the encoder/decoder of Tsukagoshi '847. 
The motivation to combine is suggested by Buchanan '490 which discloses: the automatic 
temporal formatter of Buchanan '490 operates during run-time (Buchanan '490: col. 3, Ins. 1 1- 
1 5) and further that application of the automatic temporal formatter of Buchanan '490 and 
Garbinato '97 provides the advantage of handling unpredictable data changes such as that of the 
runtime subtitle to video/audio matching of Tsukagoshi '847 (Buchanan '490: col. 3, Ins. 40-47; 
col. 6, Ins. 7-10). 

It would have been further obvious to a person having ordinary skill in the art to modify 
the Buchanan '490 and Tsukagoshi '847 combination by separating the structuring functions of 
the first and second operations into distinct aligner objects as per the Strategy design pattern of 
Garbinato '97. The motivation to accomplish said modification is suggested by Garbinato '97 
which discloses that encapsulating the aligner implementations into separate objects and 
invoking via a Strategy design pattern provides the advantages of providing both design time and 
runtime composition of aligner implementations and furthermore overcomes the limitations of an 
inheritance based implementation (Garbinato '97: p. 3, col. 2, In. 3 to p. 4, col. 1, In. 24). 
Claims 18-19: 
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Examiner notes that Claims 18-19 are the apparatus embodiment of Claims 6-7 
respectively and are rejected on the same basis. 

10. Claim 8 is rejected under 35 U.S.C. 103(a) as being unpatentable over Buchanan '490 in 
view of Garbinato '97. 
Claim 8: 

Regarding Claim 8, Buchanan '490 discloses: a system for synchronizing a realization of 
a media stream (Buchanan '490: Abstract) having a first representation synchronized with said 
realization, and at least one second representation, (Buchanan '490: col. 57, Ins. 11-13) said 
method comprising: 

a structurer configured to determine structure information for said first representation 

(Buchanan '490: col. 23, Ins. 59-65; col. 57, Ins. 20-30); 
- a structurer configured to determine structure information for said at least one second 

representation (Buchanan '490: col. 23, Ins. 59-65; col. 57, Ins. 20-30); and 

a first aligner configured to align said structure information for said first representation 

and said at least one second representation (Buchanan '490: col. 23, In. 66 to col. 24, In. 

10; col. 57, Ins. 31-50). 
However, Buchanan '490 does not explicitly disclose that the structurer for the first 
representation and the structurer for the second representation are distinct. 

Garbinato '97 discloses the well-known Strategy design pattern. Specifically, Buchanan 
'97 discloses that objects designed to handle distinct types of data and/or interactions are to be 
distinct via the Strategy design pattern (Garbinato '97: p. 1, col. 2, Ins. 14-27). 
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It would have been obvious to a person having ordinary skill in the art to modify 
Buchanan '490 by separating the structuring functions of the first and second operations into 
distinct structurer objects as per the Strategy design pattern of Garbinato '97. The motivation to 
accomplish said modification is suggested by Garbinato '97 which discloses that encapsulating 
the structurer implementations into separate objects and invoking via a Strategy design pattern 
provides the advantages of providing both design time and runtime composition of structurer 
implementations and furthermore overcomes the limitations of an inheritance based 
implementation (Garbinato '97: p. 3, col. 2, In. 3 to p. 4, col. 1, In. 24). 

1 1 . Claim 9 is rejected under 35 U.S.C. 103(a) as being unpatentable over Buchanan '490 
and Garbinato '97 in view of Tsukagoshi '847. 
Claim 9: 

Regarding Claim 9, Buchanan '490 and Garbinato '97 in combination disclose all the 
limitations of Claim 8 (supra). However, Buchanan '490 and Garbinato '97 in combination do 
not disclose: at least one Tenderer configured to render said at least one second representation, 
after being synchronized, in a form suitable for displaying as an overlayed subtitle. 

Tsukagoshi '847 discloses an encoder and decoder of subtitle information. Specifically, 
Tsukagoshi '847 discloses: at least one Tenderer configured to render said at least one second 
representation, after being synchronized, in a form suitable for displaying as an overlayed 
subtitle (Tsukagoshi '847: col. 16, Ins. 1-15). Note that Tsukagoshi '847 teaches "a rate 
controller which controls the rate that the compressed video is transferred to the multiplexer in 
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synchronism with the rate that the subtitles are sent to the multiplexer" (Tsukagoshi '847: col. 
11,1ns. 37-43). 

It would have been obvious to a person to apply the substitute the automatic temporal 
formatter of Buchanan '490 and Garbinato '97 for the rate controller with the encoder/decoder of 
Tsukagoshi '847. The motivation to combine is suggested by Buchanan '490 which discloses: 
the automatic temporal formatter of Buchanan '490 and Garbinato '97 operates during run-time 
(Buchanan '490: col. 3, Ins. 11-15) and further that application of the automatic temporal 
formatter of Buchanan '490 and Garbinato '97 provides the advantage of handling unpredictable 
data changes such as that of the runtime subtitle to video/audio matching of Tsukagoshi '847 
(Buchanan '490: col. 3, Ins. 40-47; col. 6, Ins. 7-10). 

12. Claim 10 is rejected under 35 U.S.C. 103(a) as being unpatentable over Buchanan '490, 
Garbinato '97, and Tsukagoshi '847 in view of Kim '92. 
Claim 10: 

Regarding Claim 10, Buchanan '490, Garbinato '97, and Tsukagoshi '847 in combination 
disclose all the limitations of Claim 9 (supra). Buchanan '490, Garbinato '97, and Tsukagoshi 
'847 further disclose that the realization specifies a media stream (Buchanan '490: col. 57, Ins. 
1 1-13). However, Buchanan '490, Garbinato '97, and Tsukagoshi '847 in combination do not 
explicitly disclose: system further comprising: a tree aligner configured to determine a tree 
structure for said media stream 
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Kim '92 discloses a synchronization relation tree. Specifically, Kim '92 discloses: the 
system further comprising: a tree aligner configured to determine a tree structure for said media 
stream (Kim '92: Abstract). 

It would have been obvious to a person having ordinary skill in the art to apply the 
synchronization relation tree of Kim '92 to the Buchanan '490, Garbinato '97, and Tsukagoshi 
'847 in combination. The motivation to combine is suggested by Kim '92 which discloses the 
synchronization relation tree provides for both "temporal relationship consistency" and "dynamic 
schedule completion" and further is better suited for an object-oriented implementation (Kim 
'92: p.3, In. 38 to p. 4, In. 3). 

13. Claims 1 1-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Buchanan 
'490, Garbinato '97, Tsukagoshi '847, and Kim '92 in combination in further view of the 
publication, "Detection of Target Speakers in Audio Databases," by Magrin-Chagnolleau, 
published by ICASSP, 1999 (hereafter Magrin-Chagnolleau '99). 
Claims 11-12: 

Regarding Claims 11-12, Buchanan '490, Garbinato '97, Tsukagoshi '847, and Kim '92 
in combination disclose all the limitations of Claim 10 (supra). However, Buchanan '490, 
Garbinato '97, Tsukagoshi '847, and Kim '92 in combination do not explicitly disclose: 

- (Claim 1 1) means for detecting speech and non-speech boundaries; and 

- (Claim 12) means for detecting transitions and speaker changes. 
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Magrin-Chagnolleau '99 disclose: means for detecting speech and non-speech boundaries 
and means for detecting transitions and speaker changes (Magrin-Chagnolleau '99: Abstract; 
Section 4 titled, "Detection Algorithm"). 

It would have been obvious to a person having ordinary skill in the art to apply the means 
of Magrin-Chagnolleau '99 to the Buchanan '490, Garbinato '97, Tsukagoshi '847, and Kim '92 
combination. The motivation to accomplish said application is suggested by Magrin- 
Chagnolleau '99 which discloses, the advantages of automatically detecting "useful cues to 
segment, classify, and organize" audio data using multiple speakers (Magrin-Chagnolleau '99: 
Abstract, Section 1, titled, "Introduction."). 



Conclusion 

14. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

AAP A - Examiner notes a large number of prior art cited by applicant as incorporated by reference in 
the specification. This prior art is enumerated in paragraph 1 (supra) which objects to the improper 
incorporation of "essential subject matter" by reference. Furthermore, the above algorithms and 
approaches cited anticipate the application of the applicant. 

U.S. Patent No. 5,748,187 to Kim et al. "Synchronization Control of Multimedia Objects in an MHEG 
Engine." Reference teaches greater details of synchronization in MHEG. Complements the Buchanan 
'490 reference. 

U.S. Patent No. 5,895,124 to Tsuga et al. "Optical Disc and Reproduction Device Which Can Achieve 
a Dynamic Switching of the Reproduced Content." Reference is one of many which teach storing of 
subtitles on DVDs. 
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15. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Patrick J.D. Santos whose telephone number is 703-305-0707. 
The examiner can normally be reached on M-F 8:00-4:30. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Safet Metjahic can be reached on 703-308-1436. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Patrick J.D. Santos 
May 1, 2004 
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